2 research outputs found
Macaw: The Machine Learning Magnetometer Calibration Workflow
In Earth Systems Science, many complex data pipelines combine different data
sources and apply data filtering and analysis steps. Typically, such data
analysis processes are historically grown and implemented with many
sequentially executed scripts. Scientific workflow management systems (SWMS)
allow scientists to use their existing scripts and provide support for
parallelization, reusability, monitoring, or failure handling. However, many
scientists still rely on their sequentially called scripts and do not profit
from the out-of-the-box advantages a SWMS can provide. In this work, we
transform the data analysis processes of a Machine Learning-based approach to
calibrate the platform magnetometers of non-dedicated satellites utilizing
neural networks into a workflow called Macaw (MAgnetometer CAlibration
Workflow). We provide details on the workflow and the steps needed to port
these scripts to a scientific workflow. Our experimental evaluation compares
the original sequential script executions on the original HPC cluster with our
workflow implementation on a commodity cluster. Our results show that through
porting, our implementation decreased the allocated CPU hours by 50.2% and the
memory hours by 59.5%, leading to significantly less resource wastage. Further,
through parallelizing single tasks, we reduced the runtime by 17.5%.Comment: Paper accepted in 2022 IEEE International Conference on Data Mining
Workshops (ICDMW
Towards Advanced Monitoring for Scientific Workflows
Scientific workflows consist of thousands of highly parallelized tasks
executed in a distributed environment involving many components. Automatic
tracing and investigation of the components' and tasks' performance metrics,
traces, and behavior are necessary to support the end user with a level of
abstraction since the large amount of data cannot be analyzed manually. The
execution and monitoring of scientific workflows involves many components, the
cluster infrastructure, its resource manager, the workflow, and the workflow
tasks. All components in such an execution environment access different
monitoring metrics and provide metrics on different abstraction levels. The
combination and analysis of observed metrics from different components and
their interdependencies are still widely unregarded.
We specify four different monitoring layers that can serve as an
architectural blueprint for the monitoring responsibilities and the
interactions of components in the scientific workflow execution context. We
describe the different monitoring metrics subject to the four layers and how
the layers interact. Finally, we examine five state-of-the-art scientific
workflow management systems (SWMS) in order to assess which steps are needed to
enable our four-layer-based approach.Comment: Paper accepted in 2022 IEEE International Conference on Big Data
Workshop SCDM 202